Reproducing kernel Hilbert space semantics for probabilistic programs

We propose denotational semantics for a language of probabilistic arithmetic expressions based on reproducing kernel Hilbert spaces (RKHS). The RKHS approach has numerous practical advantages, but from a semantics point of view the most important is ability to provide convergence guarantees on approximate evaluations of expressions. We present preliminary results on convergence bounds, adapting them to more general settings is still work in progress. 1. A grammar of probabilistic expressions As an example of a probabilistic programming language we consider a simple grammar of probabilistic expressions. e ::= r | v | f(e1, e2) | D(e1, e2) | let v = e1in e where r is a real number, v is a variable, f is a deterministic (measurable) function from a predefined collection, D is a probability distribution (parametrised by real numbers) from a predefined collection. Although for simplicity we only consider real-valued primitives here, we emphasize that our approach is equally applicable to other data types, such as integers or strings, as long as we can define a positive definite kernel for them. Similarly, the syntax only includes binary functions for clarity of notation, but functions of arbitrary arity could be used instead. Expressions generated by this grammar correspond to probability distributions over the set of real numbers, as long as they contain no free variables. It is straightforward to give semantics to those expressions using e.g. the probability monad, but in practice the required computation may be prohibitively expensive. We show how to derive equivalent semantics based on RKHS and how to compute it approximately, potentially with convergence guarantees. 2. Introduction to RKHS Our approach is based on the existing large body of work on RKHS (Berlinet and Thomas-Agnan 2004; Schölkopf and Smola 2001). This is a vast topic, so we only provide a short introduction to the main concepts below. The basic idea is to map points in the input space X (here X = R) to a feature space H where the relationships we are interested in have a simpler algebraic form. For example, in support vector machines (SVM) non-linear boundaries in the input space become linear in the feature space. However, the feature space is usually higher dimensional than the input space, so performing computation in it is more expensive. Working explicitly with elements of the feature space can be avoided if the feature space H is an RKHS. To construct an RKHS we start with a symmetric function k : X ×X → R satisfying the following condition: for any m ∈ N, a1, . . . , am ∈ R, and x1, . . . , xm ∈ X